RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0 - #3997

Updates the Cloud RDI guides for the RDI 2.0.0 console. The current guides describe one source and the older five-step setup. The updated guides describe several sources feeding one Redis target, the six-step creation process, and source-level management.

Ticket: RDSC-6038

Companion release notes: #3996.

Changes

• Update overview, workspace, preparation, creation, quick-start, and management pages for Settings → Add sources → Configure source → Select data → Add transformations → Review & deploy. Adding a source to a running pipeline starts at Add sources because target and settings are already defined.
• Explain separate source connectivity, credentials, collectors, datasets, and metrics; source names as configuration identifiers; job source assignments; and preservation of existing source names after upgrade. Source names in the quick start are examples, not fixed identifiers.
• Cover adding sources, pending setup, dataset and transformation edits, source Start/Stop/Reset/Delete, and whole-pipeline actions. Source reset/removal cleans internal RDI state while retaining target records; the other sources keep their internal data, although the pipeline temporarily stops during cleanup. Stopping a source pauses its data capture while other sources continue.
• State that RDI Cloud uses the Flink processor for all pipelines. Remove internal keystore and topic-prefix guidance from Cloud setup instructions. Preserve Cloud restrictions, AWS connectivity and secret preparation, sample Terraform cleanup, and sanitized rejected-record behavior.
• Add a Cloud FAQ organized into Data recovery, Multiple sources, Upgrades and maintenance, and Billing. Explain reset versus flush, the steps to reload data after a flush, retained target records, and selective deletion for a single source. Include RPU examples, hourly peak billing, stopped-source versus stopped-pipeline charges, idle usage, and billing start/stop conditions.
• Require transformation jobs to be removed or reassigned before source deletion. Correct the claim that resetting automatically deletes target data.
• Correct the quick-start Terraform variable descriptions to identify the two role ARNs and fix six existing config-reference anchors. Keep existing section anchors used by inbound links.
• Retain existing screenshots wherever the controls still apply, and use 13 Light mode preview captures for changed screens. Restore 33 references to 18 existing assets across workspace creation, preparation, pipeline creation, quick-start, and management; also reuse the new transformation-job image beside the creation steps. Existing AWS setup screenshots and shared workspace-creation content remain applicable.

Preview and screenshots

Captured from the supplied k8s-idm preview for cloud-ui PR #3050, head 0c5d44ce1ffeaef48682f909de4781cf16acfbcb, using @rdi-ui/pipeline 0.1.18.

All 13 new console screenshots were retaken in Light mode to match the existing Cloud documentation screenshots. The preview query was preserved during navigation. Screenshots use the existing default sample pipeline with one PostgreSQL and two MySQL sources. They show the workspace Sources column, dashboard, source cards/actions, configuration, dataset, transformations, metrics, settings, add-source wizard, and reset/delete confirmations. Captures exclude account chrome, billing, connection addresses, and secret ARNs. Preview URLs and environment IDs are not included in the public guides.

No source was started, stopped, reset, or deleted, and no pipeline was deployed. Reset/delete confirmations were canceled, and the pipeline and all three sources remained streaming. Start behavior and naming/assignment rules were cross-checked against the source and Core documentation; destructive behavior was not exercised.

The restored screenshots show Add pipeline, target selection, connectivity and role ARNs, credentials, TLS/mTLS fields, collector and processor properties, schema/table/column selection, missing-key warnings, key selection, Edit, and Deploy pipeline. Their image files are unchanged. The existing controls were checked against the preview and its source labels; the collector-properties dialog and dataset editing were opened without saving changes. Screenshots with obsolete wizard buttons, the old workspace source columns, or transformation settings that moved to Settings remain replaced. The old target-change confirmation remains excluded because its claim of automatic re-ingestion conflicts with current behavior.

Coordination with Core documentation

Core multi-source documentation originally reviewed in #3916 is now on main through #4029 and #4030. Release notes and migration guidance from #3996 are also merged.

Merged main at 3b233607d into this branch in 67cf59d05. Resolved the conflicts in define.md, setup.md, and quick-start.md by retaining the reviewed new-console wording: Source name identifies the source in configuration and transformation jobs. Main's display-label wording applies to the older console. All previously reviewed Cloud text and screenshots are unchanged by this merge.

Review and demo follow-up

Address the review comments and September 16 demo feedback in e727b4836.

• Apply the editorial suggestions, simplify source-name and job-assignment instructions, and remove list counts and internal configuration details requested in review.
• Keep Configure source singular to match the actual UI label. Remove the unnecessary step that asked users to repair invalid jobs before adding a source; document the job-reference check at source deletion instead.
• Separate reset, flush, restart, and deletion behavior. The new workspace reset dialog has no flush checkbox; flush is a separate operation. A target flush deletes all target data, and restart alone does not reload unchanged source records.
• Cross-check billing against the current metering implementation at ab42245b1b03fb8928b674437aec23cf1890b2e2 and the demo transcript. Examples use RPU quantities, not invented prices. Do not promise the demo's future multi-pipeline invoice layout.
• Keep the existing Light mode screenshots and all applicable screenshot references. The preview currently requires sign-in, so this follow-up does not claim a new live console verification. Current preview source reviewed: cloud-ui update for redisvl 0.13.2 #3050 at 2d728e4501a102bc317ff18135333c3984237cb1, using @rdi-ui/pipeline 0.1.29. The original screenshot revision is recorded above.
• The demo's infrastructure throughput tuning and announced source/processor limit changes are not added as public self-service instructions without checking the released Cloud configuration. Core migration and TLS breaking changes remain with the companion Core/release documentation.

Latest review follow-up

• State consistently that Cloud uses Flink, without presenting it as a version-specific default or a processor choice.
• Use a general FAQ description and explicitly explain that pipeline reset covers all sources.
• Change post-flush recovery to reset while stopped, wait for reset completion, then start. The reset API preserves the pipeline active state, and the Cloud UI allows reset while stopped. Align the management guide with this behavior.
• Retain the approved upgrade FAQ and maintenance-window link.

Validation

• Hugo 0.143.1 focused Cloud/RDI build: passed without warnings.
• Internal links, anchors, frontmatter, image references, and git diff --check: passed.
• FAQ rendered and inspected, including billing tables; RPU arithmetic checked.
• Existing Light mode screenshot files and references are unchanged in this follow-up.
• Focused Hugo 0.143.1 build after merging main at 3b233607d: passed without warnings. All local links, anchors, and screenshot references resolve. The PR diff against main passes git diff --check and remains limited to Cloud docs and screenshots.

Publication hold

Do not merge until:

• The documented RDI 2.0.0 console features are released.
• Core documentation is merged through RDSC-5610: Multi-source documentation (conflict-resolved) #4029 and RDSC-5610 Multi-source documentation fixes #4030, and the three overlapping Cloud pages are reconciled.

Screenshots are prepared now from the preview; documentation publication waits for the console release.